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A brief note on the Karhunen-Loeve expansion 


Alen Alexanderian* 
October 28, 2015 


Abstract 

We provide a detailed derivation of the Karhunen-Loeve expansion of a stochastic process. We 
also discuss briefly Gaussian processes, and provide a simple numerical study for the purpose of 
illustration. 
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1 Introduction 

The purpose of this brief note is to provide a self-contained coverage of the idea of the Karhunen-Loeve 
(KL) expansion of a stochastic process. The writing of this note was motivated by being exposed to 
the many applications of the KL expansion in uncertainty propagation through dynamical systems with 
random parameter functions; see e.g., [sHHITollHl. s ince a clear and at the same time rigorous coverage 
of the KL exapnsion is not so simple to find in the literature, here we provide a simple account of 
the theoretical basis for the KL expansion, including a detailed proof of convergence. We will see that 
the KL expansion is obtained through an interesting application of the Spectral Theorem for compact 
normal operators, in conjunction with Mercer’s theorem which connects the spectral representation of a 
Hilbert-Schmidt integral operator to the corresponding Hilbert-Schmidt kernel. 
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We begin by recalling some functional analytic basics on compact operators in Section The 
material in that section are classical and can be found in many standard textbooks on the subject; see 
e.g., [5] for an accessible presentation. Next, Mercer’s Theorem is recalled in Section Then, we 
recall some basics regarding stochastic processes in Section In that section, a basic result stating the 
equivalence of mean-square continuity of a stochastic process and the continuity of the corresponding 
autocorrelation function is mentioned also. In Section we discuss in detail KL expansions of centered 
mean-square continuous stochastic processes including a proof of convergence. Finally, in Section we 
provide a numerical example where the KL expansion of a Gaussian random field is studied. 


2 Preliminaries on compact operators 


Let us begin by recalling the notion of precompact and relatively compact sets. 

Definition 2.1. (Relatively Compact) 

Let X be a metric space; A C X is relatively compact in X, if A is compact in X. 

Definition 2.2. (Precompact) 

Let X be a metric space; A C X is precompact (also called totally bounded) if for every e > 0, there 
exist finitely many points xi,..., in A such that B{xi^ e) covers A. 

The following Theorem shows that when we are working in a complete metric space, precompactness 
and relative compactness are equivalent. 

Theorem 2.3. Let X be a metric space. If A C X is relatively compact then it is precompact. 
Moreover, if X is complete then the converse holds also. 


Then, we define a compact operator as below. 


Definition 2.4. Let X and Y be two normed linear spaces and T : X ^ Y a linear map between X 
and Y. T is called a compact operator if for all bounded sets E C X, T{E) is relatively compact in Y. 


By the above definition 2.4 if ^ c X is a bounded set, then T{E) is compact in Y. The following 


basic result shows a couple of different ways of looking at compact operators. 


Theorem 2.5. Let X and Y be two normed linear spaces; suppose T : X ^ Y, is a linear operator. 
Then the following are equivalent. 


1. T is compact. 


2. The image of the open unit ball under T is relatively compact in Y. 


3. For any bounded sequence {x^} in X, there exist a subsequence of{Txn} that converges 

in Y. 


Let us denote by B[X] the set of all bounded linear operators on a normed linear space space X\ 

B[X] = {T : X ^ X| T is a bounded linear transformation.}. 

Note that equipped by the operator norm B[X] is a normed linear space. It is simple to show that 
compact operators form a subspace of B[X]. The following result (cf. [5] for a proof) shows that the 
set of compact normal operators is in fact a closed subspace of B[X]. 
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Theorem 2.6. Let {T^} be a sequence of compact operators on a normed linear space X. Suppose 
Tn T in B[X]. Then, T is also a compact operator. 

Another interesting fact regarding compact linear operators is that they form an ideal of the ring of 
bounded linear mappings B[X]. This follows from the following basic result whose simple proof is also 
included for reader’s convenience. 

Lemma 2.7. Let X be a normed linear space, and let T and S be in B[X]. IfT is compact, then so 
are ST and TS. 

Proof Consider the mapping ST. Let {xn} be a bounded sequence in X. Then, by Theorem |2.5[ 3), 
there exists a subsequence {Tx^^} of {Tx^} that converges in X\ Tx^^ ^ G X. Now, since S 
is continuous, it follows that STxn^ that is, {STxm^} converges in X also, and so ST is 

compact. To show TS is compact, take a bounded sequence {xn} in X and note that {Sx^} is bounded 
also (since S is continuous). Thus, again by Theorem |2.5[ 3), there exists a subsequence {TSxnf,} which 
converges in X, and thus, TS is also compact. □ 

Remark 2.8. A compact linear operator of an infinite dimensional normed linear space is not invertible 
in B[X]. To see this, suppose that T has an inverse S in B[X]. Now, applying the previous Lemma, 
we get that I = TS = ST is also compact. However, this implies that the closed unit ball in X is 
compact, which is not possible since we assumed X is infinite dimensional. (Recall that the closed unit 
ball in a normed linear space X is compact if and only if X is finite dimensional.) 


2.1 Hilbert-Schmidt operators 


Let D be a bounded domain. We call a function k : D x D a Hilbert-Schmidt kernel if 


[ [ \k{x,y)\Uxdy<^, 
J D J D 


that is, k G L^{D x D) (note that one special case is when k \s a continuous function on D x D). 
Define the integral operator K on LS{D), K : u ^ Ku for u G LS{D), by 


[Ku]{x) = / k{x,y)u{y)dy. 
Jd 


( 1 ) 


It is simple to show that K \s a bounded operator on LS{D). Linearity is clear. As for boundedness, we 
note that for every u G LS{D), 


j {Ku){x) d.x = J J Hx,y)u{y)dy dx 


< 

Id ^Jd ' ^Jd 

~ 11^1 U2(j)xD) I I'^l U2(r)) < C)0. 


(Cauchy-Schwarz) 


An integral operator K as defined above is called a Hilbert-Schmidt operator. The following result 
which is usually proved using Theorem |2.6| is very useful. 

Lemma 2.9. Let D be a bounded domain in and let k e L‘^{D x D) be a Hilbert-Schmidt kernel. 
Then, the integral operator K : L‘^{D) —> L‘^{D) given by [Ku]{x) = f^k{x,y)u{y)dy is a compact 
operator. 
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2.2 Spectral theorem for compact self-adjoint operators 

Let be a real Hilbert space with inner product (•,•): x M. A linear operator T \ H ^ H \s 
called self adjoint if 

(Tx, y) = (x, Ty) , Vx, ^ G H. 

Example 2.10. Let us consider a Hilbert-Schmidt operator K on b]) as in 0 (where for simplicity 

we have taken D = [a, 6] C MJ. Then, it is simple to show that K is self-adjoint if and only if 

k{x,y) = k{y,x) on [a, 6] x [a, 6]. 

A linear operator T : H ^ H, \s called positive if {Tx,x) > 0 for all x in H. Recall that a scalar 
A G M is called an eigenvalue of T if there exists a non-zero x e H such that Tx = Xx. Note that the 
eigenvalues of a positive operator are necessarily non-negative. 

Compact self-adjoint operators on infinite dimensioal Hilbert spaces resemble many properties of 
the symmetric matrices. Of particular interest is the spectral decomposition of a compact self-adjoint 
operator as given by the following: 

Theorem 2.11. Let H be a (real or complex) Hilbert space and let T : H ^ H be a compact 
self-adjoint operator. Then, H has an orthonormal basis {e^} of eigenvectors of T corresponding to 
eigenvalues A^. In addition, the following holds: 

1. The eigenvalues Xi are real having zero as the only possible point of accumulation. 

2. The eigenspaces corresponding to distinct eigenvalues are mutually orthogonal. 

3. The eigenspaces corresponding to non-zero eigenvalues are finite-dimensional. 

In the case of a positive compact self-adjoint operator, we know that the eigenvalues are non-negative. 
Hence, we may order the eigenvalues as follows 

Ai > A 2 > ... > 0. 

Remark 2.12. Recall that for a linear operator A on a finite dimensional linear space, we define its 
spectrum (t{A) as the set of its eigenvalues. On the other hand, for a linear operator T on an infinite 
dimensional (real) normed linear space the spectrum cr(T) ofT is defined by, 

(j{T) = {A G M : T — A/ is not invertible in B[X]}, 

and cf{T) is the disjoint union of the point spectrum (set of eigenvalues), contiuous spectrum, and 
residual spectrum (see for details). As we saw in Remarka compact operator T on an infinite 
dimensional space X cannot be invertible in B[X]; therefore, we always have 0 G cr{T). However, not 
much can be said on whether X = 0 is in point spectrum (i.e. an eigenvalue) or the other parts of the 
spectrum. 


3 Mercer’s Theorem 

Let D = [a, 6] C M. We have seen that given a continuous kernel k : D x D R, we can define 
a Hilbert-Schmidt operator through Q which is compact and has a complete set of eigenvectors in 
The following result by Mercer provides a series representation for the kernel k based on 
spectral representation of the corresponding Hilbert-Schmidt operator K. A proof of this result can be 
found for example in |2]. 
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Theorem 3.1 (Mercer). Let k : D x D R be a continuous function, where D = [a, b] C M. Suppose 
further that the corresponding Hilbert-Schmidt operator K : L^{D) L‘^{D) given by 0 is postive. 

lf{Xi} and {ei} are the eigenvalues and eigenvectors of K, then for all s^t e D, 

k{s, i) = ^ Xiei{s)ei{t), (2) 

i 

where convergence is absolute and uniform on D x D. 


4 Stochastic processes 

In what follows we consider a probability space P), where f}\s a sample space, T is an appropriate 

cr-algebra on Q and P is a probability measure. A real valued random variable X on (yt^fF^P) is an 
P/S(M)-measurable mapping X : yi^P^P) (M, S(M)). The expectation and variance of a random 
variable X is denoted by, 

E [X] := [ X{u}) dP{Lo), Var [X] := E [(X - E [X])^] . 

Jn 

Lf^yt^P^P) denotes the Hilbert space of (equivalence classes) of real valued square integrable random 
variables on 

L^{n,P,P) = {X :n^R: [ \X{ujy dP{uj) < oo}. 

Jn 

with inner product, (X,y) = E [XF] = f^XYdP and norm ||X|| = (X,X)^/^ 

Let D C M, a stochastic prcess is a mapping X : D x ^ R, such that X(t^-) is measurable 
for every t ^ D\ alternatively, we may define a stochastic process as a family of random variables, 
Xt : ft R with t e D, and refer to X as {Xy^eD- Both of these points of view of a stochastic 
process are useful and hence we will be switching between them as appropriate. 

A stochastic process is called centered if E [X^] = 0 for all t e D. Let be an arbitrary 

stochastic process. We note that 

Yt = E[Yt]pXu 

where Xf = Yf — E[Yt] and {Xt}teD is a centered stochastic process. Therefore, without loss of 
generality, we will focus our attention to centered stochastic processes. 

We say a stochastic process is mean-square continuous if 

liniE[(Xt+e-X*)"] =0. 


The following definition is also useful. 

Definition 4.1 (Realization of a stochastic process). Let X : D x El ^ R be a stochastic process. For 
a fixed uj ^ El, we define X : D ^ R by X{t) = Xt{uj). We call X a realization of the stochastic 
process. 

For more details on theory of stochastic processes please consult [9|[6||7]. 
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4.1 Autocorrelation function of a stochastic process 

The autocorrelation function of a stochastic process {Xt}teD is given by Rx \ D x D ^ defined 
through 

Rx{sR) = ¥j[XsXt], sR e D. 

The following well-known result states that for a stochastic process the continuity of its autocorrelation 
function is a necessary and sufficient condition for the mean-square continuity of the process. 

Lemma 4.2. A stochastic process {Xt}te[a,b] mean-square continuous if and only if its autocorrelation 

function Rx is continuous on [aR] x [a, b]. 

Proof Suppose Rx is continuous, and note that 

E [{Xt+e -Xtf]=E [X^+,] - 2E [Xt+eXt] + E [X^] = Rx{t + e,t + e) - 2Rx {t + s,t) + Rx {t, t). 
Therefore, since Rx is continuous, 

lim E )^] = lini Rx (t -\- eR -\- e) — 2Rx {t eR) Rx (T = 0- 

That is Xt is mean-square continuous. Conversely, if is mean-square continous we proceed as follows: 
\Rx{t + e,s + v)- Rx{t, s)| = |E [Xt+eX,+,] - E | 

= E [{Xt+e - Xt){Xe+u - X,)] + E [{Xt+e - Xt)Xe] + E [(X,+,. - X,)Xt] 

< E [{Xt+e - Xt){Xe+, - Xe)] \ + |e [{Xt+e ” | + |e [(X,+, - X,)X*] 

< E [{Xt+e - Xtf] E [{Xe+, - Xef] + E [{Xt+e - Xtf/'^ E [Xf] 

+ e[(x,+,-x,)2]'/"e[x2]'/", 

where the last inequality follows from Cauchy-Schwarz inequality. Thus, we have, 

[Rx{t + e,s + v)-Rx{t,s)[<E [{Xt+e - Xtf] E [(X,+, - X,)^] 

+ E [{Xt+e - Xtf/^ E [Xf]+ E [{Xe+. - X,)2]E [X2], (3) 

and therefore, by mean-square continuity of we have that 

lim \Rx(t e, s ^ u) — RxU, s)\ = 0. □ 

(£,z.)^(0,0) ' 

5 Karhunen-Loeve expansion 

Let D C R. In this section, we assume that XiI^xf^^Misa centered mean-square continuous 
stochastic process such that X G L‘^{D x Q). With the technical tools from the previous sections, we 
are now ready to derive the KL expansion of X. 

Define the integral operator K : L‘^{D) -G L‘^{D) by 

[Ku]{s) = / k{sR)u{t)dt, k{sR) = Rx{sR), ( 4 ) 

Jd 

The following lemma summarizes the properties of the operator K. 
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Lemma 5.1. Let K : L‘^{D) L‘^{D) be as in 0. Then the following hold: 

1. K is compact. 

2. K is positive 

3. K is self-adjoint. 

Proof (1) Since the process X is mean-square continuous, Lemma [4~^ implies that k{s^t) = Rx{s^t) 


is continuous. Therefore, by Lemma 2.9 K is compact. 

(2) We need to show {Ku, u) >0 for every u G where (•, •) denotes the L‘^{D) inner product. 

{Ku^u) = J Ku{s)u{s)ds = j j k{sR)u{t) dt^u{s) ds 

= J {J E[XsXt]u{t)dt^u{s)ds 


= E 

= E 

= E 


/ / XsXtu{f)u{s) dt ds 

J D J D 

(^j^X,u{s) ds)(^j^Xtu{t)dt) 

(^J^Xtu{t)dty 


> 0 , 


□ 


where we used Fubini’s Theorem to interchange integrals. 

(3) This follows trivially from Rx{sR) = Rx{t,s) and Fubini’s theorem: 

{Ku,v) = J Ku{s)v{s)ds = J (^J k{t,s)v{s)ds^u{t)dt = {u,Kv). 

Now, let K be defined as in Q the previous lemma allows us to invoke the spectral theorem for 
compact self-adjoint operators to conclude that K has a complete set of eigenvectors {e^} in L‘^{D) 
and real eigenvalues {Ai}: 

Kci = XiCi. (5) 

Moreover, since K is positive, the eigenvalues are non-negative (and have zero as the only possible 
accumulation point). Now, the stochastic process X which we fixed in the beginning of this section is 
assumed to be square integrable on D xfl and thus, we may use the basis {e^} of L‘^{D) to expand Xt 
as follows, 

(6) 


Xt = ^Xiei{t), Xi= Xtei{f)dt 
Jd 


The above equality is to be understood in mean square sense. To be most specific, at this point we 
have that the realizations X of the stochastic process X admit the expansion 




Xi Ci 


where the convergence is in L‘^{D x Pt). We will see shortly that the result is in fact stronger, and we 
have 

■" N ^ 2" 

{Xt-Y^Xi 


lim E 

N^oo 


i=l 


= 0, 
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uniformly in D, and thus, as a consequence, we have that ^ holds for all t G D. Before proving this, 
we examine the coefficients Xi in Note that Xi are random variables on Q. The following lemma 
summarizes the properties of the coefficients Xi. 

Lemma 5.2. The coefficients Xi in satisfy the following: 

1. E [x,] = 0 


2. E [xiXj] = SijXj. 

3. Var [xi] = A^. 


Proof To see the first assertion note that 


E [xi] = E 


f Xtei{t)c 

JD 

[ [ Xt{uj)ei{t)dtdP{Lo) 

Jn J D 

f f Xt{uj)ei{t)dP{uj)dt 
J D Jq 

[ E[Xt]ei{t)dt = 0, 

JD 


(Fubini) 


where the last conclusion follows from E [Xt] = 0 (X is a centered process). To see the second assertion, 
we proceed as follows 


E [xiXj] = E 


Xsei{s)ds^(^J Xtej{t)dt^ 

E Xsei{s)Xtej(t) ds dt 

J D J D 

[ [ E[XsXt]ei{s)ej{t)dsdt 
J D J D 

J J k{s,t)ej{t) dt^ei{s) ds 

/ [Kej]{s)ei{s) ds (from®) 

JD 


= {Kej.ei) 


where again we have used Fubini’s Theorem to interchange integrals and the last conclusion follows 
from orthonormality of eigenvectors of K. The assertion (3) of the lemma follows easily from (1) and 
( 2 ): 

Var [xi] = E [{xi - E [xi])‘^] = E [xf\ = A^. □ 

Now, we have the technical tools to prove the following: 
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Theorem 5.3 (Karhunen-Loeve). Let X : D xQ ^ R be a centered mean-square continuous stochastic 
process with X e x D). There exist a basis {e^} of L‘^{D) such that for all t e D, 

oo 

Xt = in 

i=l 


where coefficients Xi are given by Xi(uj) = Xt{uj)ei{t) dt and satisfy the following. 


1. E [xi] = 0 

2. E [xiXj] = SijXj. 

3. Var [xi] = X^. 


Proof Let K be the Hilbert-Schmidt operator defined as in 0- We know that K has a complete set 
of eigenvectors {e^} in L‘^{D) and non-negative eigenvalues {A^}. Note that Xi(uj) = Xt{uj)ei{t) dt 


satisfy the the properties (l)-(3) by Lemma 5.2 Next, consider 


.{t) := E 


[Xt-J2xiei 

i=l 


The rest of the proof amounts to showing lim en{t) = 0 uniformly (and hence pointwise) in D. 

n^oo 


Suit) = E 


n 

- E 


Xje, 


i=l 


= E [Xf] - 2E 
Now, E [X|] = k{t^t) with k as in Q, 


n 

Xt'^XiCiit) 

+ E 

n 

E XiXjei{t)ej{t) 

i=l 




E 


Xt^XiBiX) 


i=l 


Through a similar argument, we can show that 

E 


Xt'^i f Xsei{s) ds)ei{t) 
i=i 

= E(/ ^[XtXs]ei{s)dsyi{t) 

i = l 

n » n n 

^ E ( / Ht,s)ei{s)ds^ei{t) = 




E XiXjei{t)ej{t) 


i=l 


Therefore, by Q, Q, and (|^ we have 

n 

£n{t) = k{t,t) 


i=l 


(7) 


(8) 

( 9 ) 
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invoking Theorem |3.1| (Mercer’s Theorem) we have 

lim Sn{t) = 0, 

n^oo 

uniformly; this completes the proof. □ 


Remark 5.4. Suppose A/. = 0 for some k, and consider the coefficient Xk in the expansion (|^. Then, 
we have by the above Theorem E [xk] = 0 and Var [xk] = Ak = 0, and therefore, Xk = 0. That is, the 
coefficient Xk corresponding to a zero eigenvalue is zero. Therefore, only Xi corresponding to postive 
eigenvalues Xi appear in KL expansion of a square integrable, centered, and mean-square continous 
stochastic process. 


In the view of the above remark, we can normalize the coefficients Xi in a KL expansion and define 

1 


fi = -^Xi. This leads to the following, more familiar, version of Theorem 

V 


5.3 


Corollary 5.5. Let X : D x Tl ^ R be a centered mean-square continuous stochastic process with 
X e X D). There exist a basis {e^} of LP‘{D) such that for all t ^ D, 


oo 

X{t,uj) = ^ in L'^{fl). 

i=l 


( 10 ) 


where are centered mutually uncorrelated random variables with unit variance and are given by, 

= [ Xt{w)ei{t)dt. 

V Xi Jd 

The KL expansion of a Gaussian process has the further property that are independent standard 
normal random variables (see e.g. mmi The latter is a useful property in practical applications; for 
instance, this is used extensively in the method of stochastic finite element [1]. Moreover, in the case 
of a Gaussian process, the series representation in (10) converges almost surely [4]. 


6 A classical example 

Here we consider the KL decomposition of a Gaussian random field X, which is characterized by its 
variance and an autocorrelation function Rx{s,t) given by, 

Rx{s,t) = cr^exp (11) 

We show in Figure [^a) a plot of Rx{sR) over [0,1] x [0,1]. 

Spectral decomposition of the autocorrelation function For this particular example, the eigen¬ 
functions ei{t) and eigenvalues can be computed analytically. The analytic expression for eigenvalues 
and eigenvectors can be found for example in mm We consider the case of = 1 and Lc = 1 in ( pTj ). 
In Figure p!][b)- (c), we show the first few eigenfunctions and eigenvalues of the autocorrelation function 
defined in jll[). To get an idea of how fast the approximation, 

N 

Rxis,t) = y^^Ke.i{s)ei{t) 

i=l 

converges to Rx{sR) we show in Figure [^the plots of Rxi^^ t) loi' ^ = 2,4,6,8. In Figure I we see 
that with N = 6, absolute error is bounded by 8 x 10“^. 
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Rx(s, t) 



Figure 1: The autocorrelation function (a); the first few eigenfunctions of the integral operator whose 
kernel is given by the autocorrelation function (b), and the corresponding eigenvalues (c). 


Simulating the random field Having the eigenvalues and eigenfunctions of Rx{t, oj) at hand, we can 
simulate the random field X(t^uj) with a truncated KL expansion, 

N 

i—\ 

As discussed before, in this case, are independent standard normal variables. In Figure [^a), we plot 
a few realizations of the truncated KL expansion of •), t G [0,1] and in Figure |^b), we show the 
distribution of X(t, •) at t = 1/2 versus standard normal distribution. For this experiment we used a 
low oreder KL expansion with N = 6 terms. 

A final note regarding practical applications of KL expansions In practice, when using KL ex¬ 
pansions to model uncertainties in mathematical models, a premature a priori truncation of the KL 
expansion could potentially lead to misleading results, because the effect of the higher order oscillatory 
modes on the output of a physical system could be significant. Also, sampling such a low-order KL 
expansion results in realizations of the random field that might look artificially smooth; see for example 
the realizations of a low-order KL expansion reported in Figurej^ In Figurej^we illustrate the influence 
of the higher order modes on the realizations of the truncated KL expansion, in the context of the same 
example: in the figure, we consider two fixed realizations of the process, and for each realization we plot 
with successively larger values of N. 
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N = 2 


N = 4 



0 0 0 0 


Figure 2: Improvements of the approximations to Rx{s^t) as the expansion order is increased. 



Figure 3: (a) The autocorrelation function Rx{s,t), (b) the approximation R^{sR) with N = 6, and 
(c) pointwise difference between Rx{sR) and R^isR) with N = Q. 
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Figure 4: (a) A few realizations of the random field X(t, •) approximated by a truncated KL expansion 
with N = 6 terms, (b) distribution of at t = 1/2 (blue) versus a standard normal distribution 

(red). 



Figure 5: Two realizations of the random field X(t, •) simulated via a truncated KL expansion. To 
see the influence of the higher order oscillations captured by higher order KL modes, we successively 
increase the truncation order N. 
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